feat: stewardship api unmock (CM-1218) by ulemons · Pull Request #4195 · linuxfoundation/crowd.dev

ulemons · 2026-06-10T18:37:12Z

Summary

Removes all mock implementations from the 4 packages public API endpoints and replaces them with real queries against the packages DB. Also lands the stewardship schema migration and a backfill script to seed one unassigned stewardship row per critical package.

Changes

Stewardship schema — migration V1781094067__stewardship-tables.sql creates stewardships, stewardship_stewards, stewardship_activity, stewardship_assessments, stewardship_findings, stewardship_remediation_actions; in v1 only stewardships is populated
Backfill script — packages_worker/src/bin/stewardship-backfill.ts + runStewardshipBackfill — cursor-based idempotent batch job that inserts one unassigned row per critical package; re-checks is_critical at insert time, safe to re-run
DAL — new data-access-layer/src/osspckgs/api.ts with 5 query functions (getPackageMetrics, getPackagesByStewardshipPurls, listPackagesForApi, getPackageDetailByPurl, getAdvisoriesByPackageId); includes LATERAL join on package_repos+repos for scorecard/security/repo fields and subquery on downloads_last_30d
Backend config — CROWD_PACKAGES_DB_* env vars mapped in custom-environment-variables.json; lazy promise based singleton in backend/src/db/packagesDb.ts (race-safe: caches the in-flight Promise, not the resolved value)
API endpoints — all 4 endpoints (GET /packages/metrics, GET /packages, POST /packages:batch-stewardship, GET /packages/detail) now read from the real packages DB; v2 fields (healthScore, lifecycle, maintainerBusFactor, openVulns, stewards) remain null by design until future enrichers populate them
Field notes — packages.impact is the renamed criticality_score (migration V1780589607); packages.dependent_count is the renamed dependent_packages_count (migration V1780394591)

Type of change

JIRA ticket

ticket

Note

Medium Risk
Changes live API data sources and response population (many fields null vs mocks); incorrect advisory patched/open logic or missing env config can break clients or runtime.

Overview
Wires the public packages/stewardship API to a dedicated packages DB instead of in-memory mocks. New CROWD_PACKAGES_DB_* config and a lazy getPackagesQx() connection are used by metrics, list, detail, and batch-stewardship handlers.

Adds osspckgs/api DAL queries (metrics, list with pagination/filters, detail by purl, advisories, batch stewardship by purls) against packages, stewardships, repos, and advisory tables. Responses still match the v1 shape but several fields are intentionally null until later enrichers (health, lifecycle, stewards, batch openVulns breakdown). List sortBy=health falls back to name; lifecycle and busFactor1Only are accepted in the API but no longer filter in the DB layer.

getPackage validates purl with Zod (pkg: prefix) and maps DB columns (e.g. impact → impact score, stewardship default unassigned). Advisory resolution is derived in SQL with a known lexicographic version limitation (TODO).

^{Reviewed by Cursor Bugbot for commit 8a2b724. Bugbot is set up for automated code reviews on this repo. Configure here.}

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

github-actions

Conventional Commits FTW!

cursor · 2026-06-10T18:38:59Z

-    if (unstewardedOnly && p.stewardship !== 'unassigned') return false
-    if (staleOnly) {
-      const lastRelease = MOCK_DETAILS[p.purl]?.general.riskSignals.lastRelease
-      if (!lastRelease || new Date(lastRelease) >= staleThreshold) return false


Ignored list query filters

Medium Severity

GET /packages still validates and echoes lifecycle and busFactor1Only, but those values are no longer passed into listPackagesForApi, so results stay unfiltered while the response filters object implies they were applied.

^{Reviewed by Cursor Bugbot for commit 2d2aa85. Configure here.}

Copilot

Pull request overview

This PR wires the public v1 Packages/Stewardship API endpoints to the real packages-db (instead of in-memory mocks) by introducing a packages DB connection helper in the backend and a new DAL module (osspckgs/api) that queries package metrics, listings, detail, advisories, and batch stewardship rows.

Changes:

Add packagesDb config/env wiring (CROWD_PACKAGES_DB_*) and a lazy getPackagesQx() connection helper.
Add new DAL query module services/libs/data-access-layer/src/osspckgs/api.ts and export it through DAL entrypoints.
Update public API handlers (/packages, /packages/metrics, /packages/detail, /packages:batch-stewardship) to fetch from the packages DB and return v1 responses (with several v2 fields intentionally null).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/osspckgs/index.ts	Re-export new OSS packages DAL API module.
services/libs/data-access-layer/src/osspckgs/api.ts	Introduce SQL queries backing metrics, list, detail, advisories, and batch stewardship lookup.
services/libs/data-access-layer/src/index.ts	Export new DAL API module from the package root.
backend/src/db/packagesDb.ts	Add lazy, singleton-style packages DB QueryExecutor initializer.
backend/src/conf/index.ts	Add `PACKAGES_DB_CONFIG` configuration entry.
backend/src/api/public/v1/packages/listPackages.ts	Replace mock listing logic with DAL-backed pagination and mapping.
backend/src/api/public/v1/packages/getPackagesMetrics.ts	Replace mock metrics with DAL-backed metrics query.
backend/src/api/public/v1/packages/getPackage.ts	Replace mock detail response with DB-backed package + advisories fetch and response mapping.
backend/src/api/public/v1/packages/batchGetStewardship.ts	Replace mock batch stewardship with DB-backed lookup keyed by PURL.
backend/config/custom-environment-variables.json	Map `packagesDb` config to `CROWD_PACKAGES_DB_*` environment variables.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+  const qx = await getPackagesQx()
+  const { rows, total } = await listPackagesForApi(qx, {
+    page,
+    pageSize,
+    ecosystem,
+    staleOnly,
+    unstewardedOnly,
+    sortBy,
+    sortDir,
  })


+    openVulns: null,
+    stewardship: (r.stewardshipStatus ?? 'unassigned') as StewardshipStatus,
+    stewards: null,
+  }))


joanagmaia · 2026-06-11T07:47:07Z

+    general: {
+      healthScore: null,
+      impact: {
+        impactScore: pkg.criticalityScore != null ? Math.round(Number(pkg.criticalityScore)) : null,


I would suggest to * 100 and then do a Math.round

joanagmaia · 2026-06-11T07:48:49Z

-  ok(res, detail)
+  const advisories = await getAdvisoriesByPackageId(qx, pkg.id)
+
+  ok(res, {


Why transitiveReach, maintainerBusFactor, resolution, securityContacts (this one we should check with Mouad how he is storing these) are null?

joanagmaia · 2026-06-11T07:49:33Z

+    maintainerBusFactor: null,
+    openVulns: null,


We could get both of these

joanagmaia · 2026-06-11T07:52:53Z

+      COUNT(*) AS total,
+      COUNT(*) FILTER (WHERE has_critical_vulnerability = true) AS critical


I don't think we should use `has_critical_vulnerability for this the critical ones. I'm a bit unsure what Nuno meant by this, but for now we could use the ones where Health is critical. Let's add a todo here to confirm this with product, and we can update later.

… filter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

cursor · 2026-06-11T10:39:59Z

+    impact: r.criticalityScore != null ? Math.round(Number(r.criticalityScore) * 100) : null,
+    lifecycle: null,
+    maintainerBusFactor: null,
+    openVulns: r.openVulns,


List openVulns wrong response shape

High Severity

GET /packages now puts a plain integer in openVulns, but the public contract and OpenVulns type expect an object with low, medium, high, and critical counts (or null). Clients that read severity fields from the list response will get wrong or missing data.

Additional Locations (1)

services/libs/data-access-layer/src/osspckgs/api.ts#L117-L118

^{Reviewed by Cursor Bugbot for commit 3066140. Configure here.}

cursor · 2026-06-11T10:39:59Z

+    SELECT
+      COUNT(*) AS total,
+      -- TODO: confirm with product whether "critical" here means health=critical, not has_critical_vulnerability
+      COUNT(*) FILTER (WHERE has_critical_vulnerability = true) AS critical


Metrics critical count wrong criterion

Medium Severity

getPackageMetrics sets criticalPackages using has_critical_vulnerability, while product review on this PR indicated that metric should reflect critical health (with confirmation pending), not the vulnerability flag. Dashboard totals can disagree with intended stewardship semantics.

^{Reviewed by Cursor Bugbot for commit 3066140. Configure here.}

cursor · 2026-06-11T10:39:59Z

+            WHEN ar.fixed_version IS NULL AND ar.last_affected IS NULL THEN FALSE
+            WHEN ar.fixed_version IS NOT NULL AND p.latest_version >= ar.fixed_version THEN TRUE
+            WHEN ar.fixed_version IS NOT NULL THEN FALSE
+            WHEN ar.last_affected IS NOT NULL AND p.latest_version > ar.last_affected THEN TRUE


Advisory resolution uses string compare

Medium Severity

getAdvisoriesByPackageId marks advisories patched or open by comparing latest_version to range bounds with SQL >= and >. That is lexicographic, not semver or Maven ordering, so patched vs open can be wrong for typical version strings.

^{Reviewed by Cursor Bugbot for commit 3066140. Configure here.}

cursor · 2026-06-11T10:39:59Z

+          pkg.downloadsLast30d != null ? parseInt(pkg.downloadsLast30d, 10) : null,
+        dependentPackages: pkg.dependentPackagesCount ?? null,
+        dependentRepos: pkg.dependentReposCount ?? null,
+        transitiveReach: pkg.transitiveReach,


transitiveReach wrong API format

Medium Severity

Package detail returns transitiveReach as a numeric percentile rank from SQL, while the public schema and previous mock responses use a human-readable string such as Top 0.4%. Consumers expecting the documented string format will misinterpret or fail to render the field.

Additional Locations (1)

services/libs/data-access-layer/src/osspckgs/api.ts#L206-L215

^{Reviewed by Cursor Bugbot for commit 3066140. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8a2b724. Configure here.}

cursor · 2026-06-11T12:09:20Z

+    },
+    provenance: {
+      repositoryMapping: {
+        declaredRepo: pkg.repoUrl ?? pkg.repositoryUrl ?? pkg.declaredRepositoryUrl ?? null,


Declared repo uses mapped URL

Medium Severity

provenance.repositoryMapping.declaredRepo prefers repoUrl from the best-confidence repo join before declaredRepositoryUrl, so the field can expose an inferred mapping instead of the package’s declared repository.

^{Reviewed by Cursor Bugbot for commit 8a2b724. Configure here.}

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

+    health: null,
+    impact: r.criticalityScore != null ? Math.round(Number(r.criticalityScore) * 100) : null,
+    lifecycle: null,
+    maintainerBusFactor: null,
+    openVulns: r.openVulns,
+    stewardship: (r.stewardshipStatus ?? 'unassigned') as StewardshipStatus,


+          pkg.downloadsLast30d != null ? parseInt(pkg.downloadsLast30d, 10) : null,
+        dependentPackages: pkg.dependentPackagesCount ?? null,
+        dependentRepos: pkg.dependentReposCount ?? null,
+        transitiveReach: pkg.transitiveReach,
+      },


+      p.latest_release_at AS "latestReleaseAt",
+      s.status AS "stewardshipStatus",
+      (SELECT COUNT(*)::int FROM advisory_packages ap WHERE ap.package_id = p.id) AS "openVulns",
+      COUNT(*) OVER() AS total
+    FROM packages p
+    LEFT JOIN stewardships s ON s.package_id = p.id


+      -- percentile rank within ecosystem (0=least, 1=most transitive reach)
+      (
+        SELECT r.prank
+        FROM (
+          SELECT purl, PERCENT_RANK() OVER (PARTITION BY ecosystem ORDER BY transitive_dependent_count ASC NULLS FIRST) AS prank
+          FROM packages
+          WHERE ecosystem = p.ecosystem
+        ) r
+        WHERE r.purl = p.purl
+      ) AS "transitiveReach"


ulemons added 2 commits June 10, 2026 16:17

feat: adding stewardship tables and small fixis

81f23a4

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: openapi refactor

b7f1724

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Copilot AI review requested due to automatic review settings June 10, 2026 18:37

Copilot started reviewing on behalf of ulemons June 10, 2026 18:37 View session

ulemons self-assigned this Jun 10, 2026

github-actions Bot reviewed Jun 10, 2026

View reviewed changes

cursor Bot reviewed Jun 10, 2026

View reviewed changes

ulemons changed the title ~~Feat/stewardship api unmock~~ feat: stewardship api unmock (CM-1218) Jun 10, 2026

Copilot AI reviewed Jun 10, 2026

View reviewed changes

joanagmaia reviewed Jun 11, 2026

View reviewed changes

ulemons and others added 2 commits June 11, 2026 10:30

fix: align stewards schema to array type and fix unstewardedOnly null…

f1421c2

… filter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

feat: add backfill script

a739c9f

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

ulemons force-pushed the feat/backfill-stewardship-script branch from 399b9ba to a739c9f Compare June 11, 2026 08:33

ulemons added 3 commits June 11, 2026 10:35

feat: add packages api

1bcf9ad

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: db visibility

f0cebf0

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add null info which can be already fetched

3066140

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

ulemons force-pushed the feat/stewardship-api-unmock branch from 2d2aa85 to 3066140 Compare June 11, 2026 10:39

cursor Bot reviewed Jun 11, 2026

View reviewed changes

ulemons force-pushed the feat/backfill-stewardship-script branch from a739c9f to 4e73636 Compare June 11, 2026 11:34

Base automatically changed from feat/backfill-stewardship-script to main June 11, 2026 11:43

Merge branch 'main' into feat/stewardship-api-unmock

8a2b724

Copilot AI review requested due to automatic review settings June 11, 2026 12:08

Copilot started reviewing on behalf of ulemons June 11, 2026 12:08 View session

cursor Bot reviewed Jun 11, 2026

View reviewed changes

Copilot AI reviewed Jun 11, 2026

View reviewed changes

		COUNT(*) AS total,
		COUNT(*) FILTER (WHERE has_critical_vulnerability = true) AS critical

Conversation

ulemons commented Jun 10, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Type of change

JIRA ticket

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 10, 2026

Choose a reason for hiding this comment

Ignored list query filters

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

joanagmaia Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 11, 2026

Choose a reason for hiding this comment

List openVulns wrong response shape

Uh oh!

cursor Bot Jun 11, 2026

Choose a reason for hiding this comment

Metrics critical count wrong criterion

Uh oh!

cursor Bot Jun 11, 2026

Choose a reason for hiding this comment

Advisory resolution uses string compare

Uh oh!

cursor Bot Jun 11, 2026

Choose a reason for hiding this comment

transitiveReach wrong API format

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 11, 2026

Choose a reason for hiding this comment

Declared repo uses mapped URL

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ulemons commented Jun 10, 2026 •

edited by cursor Bot

Loading

github-actions Bot left a comment •

edited

Loading